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Attorney Docket No.: 015358-006000US 
Client Reference No.: ID-RSV-229 

TECHNIQUES FOR TARGETING INFORMATION TO USERS 

COPYRIGHT NOTICE 
[01] A portion of the disclosure of this patent document contains material 
that is subject to copyright protection. The copyright owner has no objection to the 
5 xerographic reproduction by anyone of the patent document or the patent disclosure in 

exactly the form it appears in the U.S. Patent and Trademark Office patent file or records, but 
otherwise reserves all copyright rights whatsoever. 

p CROSS-REFERENCES TO RELATED APPLICATIONS 

'^,0 [02] This application claims priority from the following U.S. Patent 

O Applications, the entire contents of which are herein incorporated by reference for all 
Q purposes: 

^ [03] (1) U.S. Patent Application Serial No. 08/995,616, entitled, 

l_ "AUTOMATIC ADAPTIVE DOCUMENT HELP SYSTEM" (Attorney Docket No. 1 53 58- 

3 5 004200US; ID-CRC-176) filed December 22, 1997; 

t: [04] (2) U.S. Patent Application Serial No. 09/636,039, entitled, 

□ "TECHNIQLffiS TO FACILITATE READING OF A DOCUMENT" (Attorney Docket No. 
1 5358-004240US; ID-CRC-234) filed August 9, 2000; 

[05] (3) U.S. Patent Application Serial No. 09/483,092, entitled, 
20 "METHOD, SYSTEM AND COMPUTER CODE FOR CONTENT BASED WEB 

ADVERTISING" (Attorney Docket No. 15358-005900US; ID-RSV-213) filed January 14, 
2000; and 

[06] (4) U.S. Patent Application Serial No. 09/483,094, entitled, 
"METHOD FOR INTRA-DOCUMENT COMPARISON IN A CONTENT BASED WEB 
25 ADVERTISING SYSTEM" (Attorney Docket No. 15358-005910US; ID-RSV-239) filed 
January 14, 2000. 



BACKGROUND OF THE INVENTION 
[07] The present invention relates to providing and outputting information 
30 to a user in electronic form, and more particularly to techniques for targeting electronic 
information to a user based on contents of document(s) viewed or accessed by the user. 




[08] With the rapid growth of computers and communication networks such 
as the Intemet, an increasing amount of information is now available to users in electronic or 
digital form. The information is usually stored by a communication network in the form of 
documents that can be accessed by a user of the communication network using computer 
5 systems coupled to the communication network. 

[09] For example, in the World Wide Web (WWW) environment, the 
information is stored in the form of hypertext documents called web pages that can be 
accessed by a user using a browser program executing on a client computer coupled to the 
Internet. A web page may incorporate several information objects storing information of 
^0 various types. These information objects may include text objects, graphics objects, audio 
^ and video content objects, multimedia objects, software program objects, hypertext link 
□ objects, and other types of data objects and/or combinations thereof. The hypertext link 
"Zl objects may contain hypertext links to other web pages. Web pages are typically stored on 
y web servers or content servers coupled to the Internet. Each web page is uniquely identified 
5 1 5 by an address called a Uniform Resource Locator (URL) that enables users to access the web 
5 page. 

[10] The terms "client" and "server" are used to classify computer systems 
O connected to communication networks such as the Intemet based upon the role that the 
^ computer systems play with respect to requesting information or providing information. A 
20 computer system that is used by a user to request information is referred to as a "client" 
computer (or a "client"). A computer system that stores information and provides the 
information in response to an information request received fi"om a client computer is referred 
to as a "server" computer (or a "server"). A particular computer system may function both as 
a client and as a server. 

25 [11] Users typically access web pages using a program called a "web 

browser" which generally executes on a client computer coupled to the Intemet. A web 
browser is a type of client application that enables users to select, retrieve, and perceive 
information stored by the Intemet. Examples of browsers include the Intemet Explorer 
browser program provided by Microsoft Corporation, the Netscape Navigator browser 

30 provided by Netscape Corporation, and others. Users generally access web pages by 

providing URL information to the browser, either directly or indirectly, and the browser 
responds by retrieving the web page corresponding to the user-provided URL. The retrieved 
web page is then displayed to the requesting user on the client computer. 
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[12] Due to the vast volume of information available via communication 
networks such as the Intemet, it is becoming increasingly difficult for a user to identify 
documents that contain information of interest to the user or to find information that is 
relevant to the user. While there are several tools (e.g. search engines, etc.) available which 
5 help the user in finding relevant information, each of these tools requires manual input firom 
the user and also requires that the user know how to use the tool. 

[13] For example, in order to use a search engine, the user has to know how 
to formulate an appropriate search query that will be used by the search engine to identify 
documents of interest to the user. Formulating an appropriate search query can be a non- 
_10 trivial task, especially for those users who are not comfortable with using computers or who 

are not computer sawy. Further, if the search query is not properly formulated, the results 
S that are retumed by the search engine might not suit the user's needs. For example, if the 
yl search query is too broad, the number of documents retumed by the search engine might be 
yj quite large and include documents that are irrelevant to the user. The user is then forced to 
waste valuable time in identifying relevant documents from the vast number of documents 
O retumed by the search engine. Conversely, if the search query is too narrow, then the search 
engine may miss documents that are relevant to the user. 

[14] In light of the above, what is needed are techniques for providing 
H' relevant information to users without requiring specific user input. It is further desired that 
20 the techniques be able to provide relevant information to a user based upon information or 
documents accessed by the user and based upon the user's likes and preferences. 

BRIEF SUMMARY OF THE INVENTION 
[15] The present invention provides techniques for providing relevant 
25 information to a user. According to an embodiment of the present invention, the relevant 

information is selected based on contents of information or documents accessed or viewed by 
the user. The relevant information is provided automatically to the user without requiring 
user input. The information provided to the user is selected fi"om information provided by 
various content provider systems. The present invention uses several techniques to determine 
30 relevancy between the contents of the document accessed by the user and between the 

information provided by the content provider systems. The relevancy information is used to 
select information to be provided to the user. 

[16] According to an embodiment, the present invention provides 
techniques for providing information to a user based upon contents of a first document 
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displayed to the user. According to this specific embodiment, the present invention identifies 
the first document displayed to the user, identifies at least a first section of the first document, 
extracts a first set of information objects from the first section of the first document, 
determines degree of relevancy information for a second set of information objects, the 
5 degree of relevancy information indicating the relevancy of information objects in the second 
set of information objects to information objects in the first set of information objects, and 
selects a third set of information objects from information objects in the second set of 
information objects based upon the degree of relevancy information determined for 
information objects in the second set of information objects, wherein information objects in 

J^O the third set of information objects store information to be output to the user when the first 

yQ document is being displayed to the user. 

g [17] According to another embodiment of the present invention, a system 

^ comprising a user system and a server system is provided for providing information to a user 
U based upon contents of a document displayed to the user. In this embodiment, the user 
^iS system is used to display the first document to the user. The server system is configured to 

access a first set of content provider information objects (CPIOs), identify the document 
M= displayed to the user, extract a first set of user docimient information objects (UDIOs) fi*om 

the document, and identify a plurality of selection techniques for determining degree of 
^ relevancy information for the first set of CPIOs. For each selection technique in the plurality 
20 of selection techniques, the server system is configured to apply the selection technique to 

generate degree of relevancy information for the CPIOs, the degree of relevancy information 

indicating the relevancy of the CPIOs to the UDIOs calculated using the selection technique. 

The server system is then configured to select a second set of CPIOs fi*om the first set of 

CPIOs based upon the degree of relevancy information for the CPIOs calculated using the 
25 plurality of selection techniques. The user system is configured to output information stored 

by the second set of CPIOs to the user. 

[18] Various additional objects, features and advantages of the present 

invention can be more fiilly appreciated with reference to the detailed description and 

accompanying drawings that follow. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 
[19] Fig. 1 is a simplified block diagram of a distributed system for 
targeting information to users according to an embodiment of the present invention; 



4 




[20] Fig. 2 is a simplified block diagram of a computer system according to 
an embodiment of the present invention; 

[21] Fig, 3 is a simplified high-level flowchart showing a method of 
providing targeted information to the user according to an embodiment of the present 
5 invention; 

[22] Fig. 4 is a simplified high-level flowchart showing processing 
performed for extracting UDIOs according to an embodiment of the present invention; 

[23] Fig. 5 is a simplified high-level flowchart showing processing 
performed for selecting CPIOs to be provided to the user according to an embodiment of the 
JO present invention; 

[24] Fig. 6 is a simplified high-level flowchart showing a method of 
□ selecting CPIOs using a class-based selection technique and a concept-based selection 
pi technique according to an embodiment of the present invention; 

W [25] Fig. 7 is a simplified block diagram showing the inputs and outputs of 

= 15 a source-based combination technique according to an embodiment of the present invention; 
rj [26] Fig. 8 is a simplified high-level flowchart showing processing 

^ performed for generating a single "final" list of CPIOs according to an embodiment of the 
Q present invention; and 

[27] Figs. 9A, 9B, and 9C are examples of user interfaces which may be 
20 used to output information stored by the selected CPIOs to the user according to 
embodiments of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 
[28] The present invention provides techniques for providing relevant 

25 information to users. In particular, the present invention provides techniques for providing 
relevant information (or targeting information) to a user based on information or documents 
accessed or viewed by the user. According to the teachings of the present invention, the 
relevant information is provided automatically to the user without requiring user input. 
Providing or targeting relevant information (or information in which the user is likely to be 

30 interested in) to the user involves the organized selection of information to be provided to the 
user based upon information accessed or viewed by the user and outputting the selected 
information to the user. The information selected and output to the user may be of various 
types including electronic text information, news information, advertising information, public 
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information, audio information, video information, multimedia information, images, and 
other types of information and combinations thereof. 

[29] Fig. 1 is a simpUfied block diagram of a distributed system 100 for 
targeting information to users according to an embodiment of the present invention. As 
5 shown in Fig. 1, system 100 comprises a plurality of computer systems including one or more 
content provider systems (CPSs) 104, a server system 1 12, and one or more user systems 102. 
Server system 112 and CPSs 104 are coupled to communication network 106 via 
communication links 108. A database 110 may also be locally or remotely coupled to server 
112. User systems 102 may be coupled directly to communication network 106 (e.g. user 
10 system 102-3) via communication links 108 or may alternatively be coupled to a server 
5 system 112 via communication links 108 (e.g. user systems 102-1 and 102-3). 

[30] Distributed system 100 depicted in Fig. 1 is merely illustrative of an 
iH embodiment incorporating the present invention and does not limit the scope of the invention 
jrl as recited in the claims. One of ordinary skill in the art would recognize other variations, 
-15 modifications, and alternatives. 

0 [31] Communication network 106 provides a mechanism allowing the 

i2 various computer systems depicted in Fig. 1 to commimicate and exchange information with 

1 each other. Communication network 106 may itself be comprised of many interconnected 
computer systems and communication links. While in one embodiment communication 

20 network 106 is the Internet, in other embodiments, communication network 106 may be any 
suitable computer network including a local area network (LAN), a wide area network 
(WAN), a wireless network, an intranet, a private network, a public network, a switched 
network, and the like. 

[32] Communication links 108 may be hardwire links, optical links, satellite 

25 or other wireless communications links, wave propagation links, or any other mechanisms for 
communication of information. Various communication protocols may be used to facilitate 
communication between the various systems shown in Fig. 1 . These communication 
protocols may include TCP/IP, HTTP and HTML protocols, extensible markup language 
(XML), wireless application protocol (WAP), vendor-specific protocols, customized 

30 protocols, and others. 

[33] Users may use user systems 102 to request and view information 
stored within distributed network 100. The term "users" as used in this application may refer 
to one or more human beings interacting with an user system 102, one or more processes 
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executing on user system 102, systems or devices coupled to user systems 102, and other like 
entities which may access information via user system 102. 

[34] As indicated above, the information stored within distributed network 
100 may be organized in the form of documents and the user may access the documents via 
5 user system 102. For example, in the WWW environment, the information is organized in 
the form of web pages that may be accessed by the user via a browser program executing on 
user system 102. Several different types of documents may be accessed by the user including 
documents created using word processor programs, multimedia documents, spreadsheets, 
electronic mail (email) documents, text documents, images or graphics file, audio files, video 
10 files, or other types of documents and combinations thereof These documents may be 
O accessed by the user via programs such a word processor, an email program, a text editor, a 

browser, and other like applications executing on user system 102. In general, the term 
p "access program" will be used throughout this application to refer to a program or application 
O which executes on user system 102 (or on devices coupled to user system 102) and which is 
gl5 used by a user of user system 102 to access documents or other information stored locally by 

user system 102 or stored by the various components of distributed system 100. 
y- [35] Information targeted for a user according to the teachings of the 

fy present invention may also be output to the user via user system 102. According to an 
^ embodiment of the present invention, the targeted information is output to the user without 
20 requiring any specific user input or interaction. The selected information may be presented to 
the user using several different techniques. According to an embodiment of the present 
invention, the selected information is presented to the user in a specially designated area 
(hereinafter referred to as the "selected information display area") of an access program 
executing on user system 102. One or more "selected information display areas" may be 
25 provided on an access program for displaying or outputting the selected information. The 
selected information display areas provide consistent, unobtrusive access to the selected 
information. The selected information may also be output to the user via audio and/or video 
output devices coupled to user system 102. 

[36] A user system 102 typically operates as a client requesting information 
30 fi-om servers coupled to distributed network 100. It should however be apparent that a 

particular user system 1 02 may act both as a client or a server depending on whether the user 
system is requesting or providing information. 

[37] Content provider systems 104 are configured to store information that 
may be requested and accessed by users of user systems 102. As indicated above, content 
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provider systems 104 may store the information in the form of documents. For example, 
content provider systems 104 may host w^eb sites that may store web pages that may be 
accessed by the users. In a specific embodiment of the present invention, a content provider 
system 104 may receive a web page request from a user system, and in response, the content 
5 provider system may determine the requested web page and forward the requested web page 
to the requesting user system 102. For user systems coupled to communication network 106 
via server system 112, the request web page may be communicated to the requesting user 
system via server system 112. 

[38] According to the teachings of the present invention, content provider 
10 systems 104 may also provide information (hereinafter referred to as "selection base 
^ information") from which the information to be targeted to a particular user is selected. 

According to an embodiment of the present invention, the selection base information is stored 
p by the content provider systems and made accessible to server 112 that performs the selection 
%l process. Altematively, the selection base information may be communicated by the content 
Wl5 provider systems to server 112 and stored by server 112. The selection base information may 
^ be stored and communicated to server system 1 12 in various forms such as a digital 
^ document, as a data stmcture, as a database, etc. 

[39] According to an embodiment of the present invention, server system 
Q 100 is configured to perform processing to select information from the selection base 
"^0 information to be targeted to users of user systems 1 02 based upon information accessed or 
viewed by the users. As indicated above, the user systems may be directly coupled to 
communication network 106 or may be coupled via communication network 106 via server 
system 112. For user systems that are coupled to the communication network via a server 
system, the selection of information may be performed by the server system. For user 
25 systems which are not coupled to a server system (e.g. user system 102-3), the information 
selection may be performed by the user system itself (computer system acts both as a user 
system and as a server system) or by a server system remotely coupled to the user system via 
communication network 106. 

[40] In order to facilitate selection of the information to be provided to 
30 users, server system 1 12 is configured to monitor information or documents accessed or 

viewed by the users. Based upon the document(s) viewed/accessed by the users, server 112 
selects information that is likely to be relevant to the users from selection base information 
provided to or made accessible to server system 112. 
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[41] Server system 112 may use several different techniques to determine 
and select information which may be of interest to a particular user and which is to be 
provided to the particular user. According to a first technique, the selection may be 
performed using a "concept-based" selection technique wherein server system 112 
5 determines if the contents of the document(s) viewed/accessed by the particular user are 
relevant to concepts of interest to the user and then uses the relevant concepts to determine 
information from the selection base information to be provided to the user. 

[42] Information related to concepts of interest for a particular user may be 
provided by the user and stored in a user profile file accessible to server system 112. In 
10 alternative embodiments of the present invention, the user profile information may be 

automatically generated by server system 112 based upon the particular user's interactions 
y with user system 102 and based upon information accessed by the user via user system 102. 
^ For example, server 112 may analyze the URLs associated with web pages accessed by the 
J user via user system 1 02 to determine information related to web sites accessed by the user 
j-f 5 and use that information to determine concepts of interest to the user. Server 112 may also 
gi analyze the contents of web pages accessed by the user. Server 112 may also monitor "web 
channels" that a user pemses and use that information to build a user profile. Other 
information related to the particular user accessible to server system 1 12, e.g., demographic 
f-y information related to user, etc., may also be used to build a user profile for the particular 
rio user, 

[43] According to another technique, server system 112 compares the 
contents of the document(s) viewed/accessed by the user with the contents of the selection 
base information and determines information from the selection base information to be 
provided to the user based upon the results of the comparison. Several other techniques may 

25 also be used by server system 1 12 to select information to be presented to the user. Server 
system 112 may also use a plurality of selection techniques and then combine the results of 
the plurality of selection techniques to determine information to be presented to the user. 
Further details related to techniques used by server system 1 12 to select information to be 
provided to the user are described below, 

30 [44] Information used by server system 1 12 for the information selection 

process may be stored in database 110 that is coupled either locally or remotely to server 112. 
For example, information 1 10-a related to concepts of interest to a user, information related 
to documents viewed by the user 1 10-b, selection base information 1 10-c, etc. may be stored 
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in database 110. Other information 1 10-d used by server 112 may also be stored by server 
112, 

[45] Fig. 2 is a simplified block diagram of a computer system 200 
according to an embodiment of the present invention. Computer system 200 may be used as 
5 user (or client) system 102, a content provider system 104, a server system 112, and any other 
system included in distributed network 100. As shown in Fig. 2, computer system 200 
includes at least one processor 202, which communicates with a number of peripheral devices 
via a bus subsystem 204. These peripheral devices may include a storage subsystem 206, 
comprising a memory subsystem 208 and a file storage subsystem 210, user interface input 
10 devices 212, user interface output devices 214, and a network interface subsystem 216. The 
^ input and output devices allow user interaction with computer system 200. As previously 
indicated, a user may be a human user, a device, a process, another computer, and the like. 
~ Network interface subsystem 216 provides an interface to other computer systems and to 
yl communication network 106. 

hi 5 [46] Bus subsystem 204 provides a mechanism for letting the various 

~ " components and subsystems of computer system 200 communicate with each other as 
P intended. The various subsystems and components of computer system 200 need not be at 
u the same physical location but may be distributed at various locations within network 100. 

Although bus subsystem 204 is shown schematically as a single bus, altemative embodiments 
1=^0 of the bus subsystem may utilize multiple busses. 

[47] User interface input devices 212 may include a keyboard, pointing 
devices such as an optical tablet, a mouse, trackball, touchpad, a graphics tablet, a scanner, a 
barcode scanner, a touchscreen incorporated into the display, audio input devices such as 
voice recognition systems, microphones, and other types of input devices. In general, use of 
25 the term "input device" is intended to include all possible types of devices and ways to input 
information using computer system 200. 

[48] User interface output devices 214 may include a display subsystem, a 
printer, a fax machine, or non- visual displays such as audio output devices. The display 
subsystem may be a cathode ray tube (CRT), a flat-panel device such as a liquid crystal 
30 display (LCD), or a projection device. The display subsystem may also provide non-visual 
display such as via audio output devices. In general, use of the term "output device" is 
intended to include all possible types of devices and ways to output information from 
computer system 200. 

10 



[49] Storage subsystem 206 may be configured to store the basic 
programming and data constructs that provide the functionaHty of the present invention. For 
example, according to an embodiment of the present invention, software modules 
implementing the functionality of the present invention may be stored in storage subsystem 
206. These software modules may be executed by processor(s) 202 of computer system 200. 
In a distributed environment, the software modules may be stored on a plurality of computer 
systems and executed by processors of the plurality of computer systems. Storage subsystem 
206 may also provide a repository for storing various databases that may be used by the 
present invention. Storage subsystem 206 may comprise memory subsystem 208 and file 
storage subsystem 210. 

[50] Memory subsystem 208 may include a number of memories including 
a main random access memory (RAM) 218 for storage of instructions and data during 
program execution and a read only memory (ROM) 220 in which fixed instructions are 
stored. File storage subsystem 210 provides persistent (non- volatile) storage for program and 
data files, and may include a hard disk drive, a floppy disk drive along with associated 
removable media, a Compact Digital Read Only Memory (CD-ROM) drive, an optical drive, 
removable media cartridges, and other like storage media. One or more of the drives may be 
located at remote locations on other connected computers. 

[51] Computer system 200 itself can be of varying types including a 
personal computer, a portable computer, a workstation, a computer terminal, a network 
computer, a mainframe, a kiosk, a personal digital assistant (PDA), a communication device 
such as a cell phone, or any other data processing system. Due to the ever-changing nature of 
computers and networks, the description of computer system 200 depicted in Fig. 2 is 
intended only as a specific example for purposes of illustrating the preferred embodiment of 
the computer system. Many other configurations of a computer system are possible having 
more or fewer components than the computer system depicted in Fig. 2. Computer system 
200 may function as a client or a server, or combinations thereof. 

[52] Fig. 3 is a simplified high-level flowchart 300 showing a method of 
providing targeted information to the user according to an embodiment of the present 
invention. Flowchart 300 depicted in Fig. 3 is merely illustrative of an embodiment 
incorporating the present invention and does not limit the scope of the invention as recited in 
the claims. One of ordinary skill in the art would recognize other variations, modifications, 
and alternatives. According to an embodiment of the present invention, server 112 performs 
steps 302, 302, 306, 308, 310, and 312, and user.system 102 performs step 314. Software 
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code modules executed by server 112 may implement the method. Hardware modules of 
server 112 and user system 102 in conjunction with the software modules may also 
implement the method. 

[53] As shown in Fig. 3, the method is initiated when server 112 identifies a 
5 document (or documents) that is accessed or view by a user of user system 102 (step 302). 
As previously indicated, the document may be of different formats including a web page, a 
word-processor document, an email message, a spreadsheet, and the like. The document may 
be accessed or viewed by the user using an access program executing on user system 102. 
For the embodiment depicted in Fig. 3, and for sake of simplicity, the invention has been 
1 0 described assuming that the selection is based upon a single document being viewed or 

accessed by a user using user system 102. However, the scope of the present invention is not 

Q restricted to analysis of a single document. According to the teachings of the present 

invention, the selection of information may be performed based on analysis of a plurality of 

^ documents or information viewed or accessed by the user presently and in the past. 

05 [54] After identifying the document viewed or accessed by the user, server 

J system 1 12 then determines one or more sections of the document to be analyzed (step 304). 
Server system 112 may be configured to analyze the entire document, including sections of 

Sj the document not viewed by the user, or altematively, server system 112 may be configured 
to analyze specific sections of the document. According to an embodiment of the present 

^0 invention, server system 112 may be configured to only analyze the section of the document 
that is viewed by the user using the access program. Altematively, server system 112 may be 
configured to analyze other sections of the document such as the title section of the 
document, the summary section of the document, specific headings or sub-headings within 
the document, specific paragraphs within the document, and the like, and combinations 
25 thereof. According to an embodiment of the present invention, the sections of the document 
to be analyzed are user configurable. 

[55] Server system 112 then extracts information objects from the section(s) 
of the document determined in step 304 (step 306). In general, the term "information object" 
is meant to refer to an entity (e.g. a data structure, an object, etc.) which stores information of 
30 a particular type. The types of information may include text, audio, video, images, and the 
like. For example, a text information object stores information of type text, an audio 
information object (e.g. a MP3 object) stores audio information, a video information object 
(e.g. a MPEG video object) stores video information, an image information object (e.g. a GIF 
object, a JPEG object) stores image information, and the like. 

12 




[56] One or more information objects may be embedded in the document 
viewed or accessed by the user. For example, information objects which may be embedded 
in a web page may include text objects, graphics objects, audio and video content objects, 
multimedia objects, soflw£ire program objects, hypertext link objects, and the like. As part of 
5 the processing performed in step 306, server system 112 extracts information objects 
embedded in the section of the document to be analyzed as determined in step 304. 
Alternatively, as part of the processing performed in step 306, server 112 may extract the 
contents of the section of the document to be analyzed and build one or more information 
objects of the appropriate type based on the extracted content information. For purposes of 
10 this application, an information object extracted from a document viewed or accessed by a 

user or generated based upon content extracted from a document viewed/accessed by the user 
O will be referred to as a "user document information object" (or UDIO in short). The UDIOs 

extracted in step 306 may be stored in database 110 accessible to server system 1 12. 
Ti [57] Server system 112 also extracts (or generates) information objects 

Q5 from the selection base information provided by content provider systems 104. An 
m information object extracted from the selection base information or generated based upon 

content from the selection base information will be referred to as a "content provider 
SI information object" (or CPIO in short). Extraction of CPIOs may be performed when server 
py system receives the selection base information from one or more content provider systems. 
T'jlO In alternative embodiments, the content provider systems may provide the CPIOs to the 
server system, in which case server system 112 does not have to perform the extraction. 
According to an embodiment of the present invention, the CPIOs are stored in database 110 
coupled to server system 112. In altemative embodiments of the present invention, the 
CPIOs may be stored by the content provider systems and accessed by server system 112 via 
25 communication network 106. 

[58] The UDIOs extracted in step 306 are then analyzed to determine 
information related to the contents of the extracted UDIOs (step 308). Based upon 
information obtained from analyzing the UDIOs in step 308 and based upon information 
related to CPIOs accessible to server system 112, server 112 then selects one or more CPIOs 
30 to be provided to the user (step 310). Server system 1 12 may employ several different 

selection techniques to select the CPIOs in step 310. Further details related to techniques 
employed in specific embodiments of the present invention are provided below. 

[59] The CPIOs selected in step 310 are then communicated to user system 
102 (step 312). Various communication protocols may be used to communicate the selected 
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CPIOs to the user system. The information stored by the selected CPIOs is then output to the 
user at the user system (step 314). As discussed above, various techniques may be used to 
output the information contained in the selected CPIOs to the user. According to an 
embodiment of the present invention, the information contained in the selected CPIOs is 
5 output to the user in one or more "selected information display areas" provided on an access 
program which the user uses to view/access the document identified in step 302. The 
selected information may also be output to the user via audio and/or video output devices 
coupled to user system 102. 

[60] Fig. 4 is a simplified high-level flowchart showing processing 
10 performed in step 308 of Fig. 3 according to an embodiment of the present invention. Fig. 4 
is merely illustrative of an embodiment incorporating the present invention and does not limit 
the scope of the invention as recited in the claims. One of ordinary skill in the art would 
£ recognize other variations, modifications, and altematives. 

yi [61] As shown in Fig. 4, server system 1 12 determines the "type" or "class" 

H5 of each UDIO extracted (or generated) in step 306 of Fig. 3 (step 402). The class or type of 
W an information object depends on the type of information contained or stored by the 
Q information object. For example, a UDIO which stores or contains text information is of type 
'''f "text", a UDIO which stores or contains audio information is of type "audio", a UDIO which 
fy , stores or contains video information is of type "video", a UDIO which stores or contains 
QO image information is of type "image", and so on. The type or class of an information object 

thus indicates the type of content or information stored or contained by the information 

object. 

[62] For each UDIO, server system 112 then determines content recognition 
techniques to be applied to the UDIO based upon the class of the UDIO determined in step 

25 402 (step 404). For each UDIO, the content recognition techniques determined for the UDIO 
in step 404 are then applied to the UDIO to determine the contents of the UDIO (step 406). 
Various types of content recognition techniques may be used based upon the type of a UDIO. 
For example, for an audio UDIO (e.g. a sound clip), voice recognition techniques (e.g. 
ViaVoice voice recognition program) may be applied to generate a text transcript 

30 corresponding to the sound clip. Content recognition techniques applied to an audio UDIO 
may also determine attributes of the sound source(s) in the audio UDIO. For example, the 
content recognition techniques may determine the identity of the speaker(s) in the soimd clip. 
Content recognition techniques applied to a video UDIO (e.g. a video clip) may determine 
keyframes from the video. Audio recognition techniques may also be applied to a video 
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UDIO to generate a text transcript corresponding to the audio associated with the video. 
Other content recognition techniques may determine the contents of the video and the context 
of the events depicted by the video. For example, a particular content recognition technique 
may determine the entities depicted in the video clip (e.g. determine that the entities are 
human, animals, machines, etc. and information associated with the entities) and the context 
of the video, e.g. identify the locations depicted in the video object (e.g. geographical 
locations such as London, Paris, The White House, a field, a battlegroimd, a specific opera, 
etc.), time line for the information included in the video UDIO (e.g. Year 2000 U.S. 
Elections, the 1999 Super Bowl, the dinosaur age, etc.), and other contexts. For image 
objects, optical character recognition (OCR) techniques may be used to determine the 
contents of the image. For text objects, a copy of the textual information may be obtained. 
Various other content recognition techniques may also be used to determine the contents of 
the UDIOs. 

[63] For each extracted UDIO, the type information associated with the 
UDIO (determined in step 402), and information related to the contents of the UDIO 
(determined in step 406) is stored for future processing (step 408). According to an 
embodiment of the present invention, the information is stored in database 110 coupled to 
server system 112. Processing then continues with step 310 in Fig. 3. 

[64] Server system 112 may also use the processing steps shown in Fig. 4 to 
determine the contents of CPIOs extracted from (or generated fi"om) the selection base 
information. This processing may be performed when the server system receives the CPIOs 
or when the CPIOs are extracted fi-om the selection base information. For each CPIO, the 
type information associated with the CPIO, and information related to the contents of the 
CPIO may be stored in database 110 coupled to server system 1 12 for future processing. 

[65] Fig. 5 is a simplified high-level flowchart showing processing 
performed in step 310 of Fig. 3 according to an embodiment of the present invention. As 
shown in Fig. 5, server 112 identifies one or more selection techniques to be used for 
selecting the CPIOs to be provided to the user (step 502). Several different selection 
techniques may be used in accordance with the present invention. According to a specific 
embodiment of the present invention, a "concept-based" selection technique and an "object 
class-based" selection technique are used to identify CPIOs to be presented to the user. 
Details related to the concept-based and class-based selection techniques are provided below. 
According to specific embodiments of the present invention, the user of the present invention 
may specify and/or configure the selection techniques to be used for selection of CPIOs. 
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[66] Server 112 then applies each selection technique identified in step 502 
to determine relevancy between the CPIOs and the UDIOs for each selection technique (step 
504). The results of the various selection techniques applied in step 504 are then combined to 
generate a single "final" list of one or more selected CPIOs objects to be communicated to 
user system 102 (step 506). The single "final" list of selected CPIOs is then communicated 
to user system 102 according to step 312 in Fig. 3. 

[67] In altemative embodiments of the present invention, instead of 
combining the results from the various selection techniques into one single "final" list of 
CPIOs, server system 112 may be configured to communicate the CPIOs selected using the 
various selection techniques to user system 102. In this embodiment, for each selection 
technique, server system 112 communicates a Ust of selected CPIOs to the user system along 
with information identifying the selection technique used for selecting the CPIOs. User 
system 102 may then output information contained by CPIOs in the various lists received 
fi-om the server system along with information indicating the technique used for selecting the 
CPIOs in the lists. 

[68] According to an embodiment of the present invention, the user is 
allowed to select whether the CPIOs lists generated in step 504 by applying the various 
selection techniques are to be combined (as shown in Fig. 5) or whether the lists themselves 
are to be communicated to user system 102. In altemative embodiments of the present 
invention, the user may also be allowed to selectively chose which lists firom those generated 
in step 504 are to be communicated to user system 102. 

[69] As indicated above, according to a specific embodiment of the present 
invention, a "class-based" selection technique and a "concept-based" selection technique are 
used for selecting CPIOs to be communicated to user system 102. Fig. 6 depicts a simpHfied 
high-level flowchart 600 showing a method of selecting CPIOs using a class-based selection 
technique and a concept-based selection technique according to an embodiment of the present 
invention. Flowchart 600 depicted in Fig. 6 is merely illustrative of an embodiment 
incorporating the present invention and does not limit the scope of the invention as recited in 
the claims. One of ordinary skill in the art would recognize other variations, modifications, 
and altematives. Steps 602, 604, and 606 depict processing performed according to the 
concept-based selection technique, while steps 610 and 612 depict processing performed 
according to the class-based selection technique. 

[70] As shown in Fig. 6, the concept-based selection technique is initiated 
when server system 112 determines concepts of interest to the user (step 602). As indicated 
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above, information related to concepts of interest to the user (or "user concepts information") 
may be stored in a user profile file accessible to server system 112. A concept identifier may 
identify each concept of interest to a user. One or more keywords or key phrases may be 
associated with each concept (as identified by the concept identifier). A probability value 
5 may be associated with each keyword or key phrase indicating the probability that the 

concept as identified by the concept identifier is discussed given the presence of the keyword 
or key phrase. For example, a user may be interested in a concept called "Football" and the 
keywords associated with the concept may include "San Francisco 49ers," "Joe Montana," 
"Super Bowl," and the like. Probability values may be associated with the keywords. 
10 [71] As described above, according to an embodiment of the present 

O invention, a concept of interest to the user may be represented as follows: 

Concept = (Concept ^Identifier, <Keywords_and _probabilities>) 
™ where: 

O Concept _Identifier = identifies the concept; and 

SS <Keywords_and j}robabilities> = is a vector of keywords (or key phrases) 

and probability values associated with the keywords indicating the probability that the 
'"---i concept as identified by the concept identifier is discussed given the presence of the keyword. 
hi [72] A particular concept may be further divided in sub-concepts that may 

J=-' in tum be divided in sub-sub-concepts, and so on. According to a specific embodiment of the 
20 present invention, a Bayesian belief network may be used to represent user concepts 

information. For a description of a specific embodiment of a Bayesian network, please refer 
to U.S. Patent Application Serial No. 08/995,616, entitled, "AUTOMATIC ADAPTIVE 
DOCUMENT HELP SYSTEM" (Attorney Docket No. 15358-004200US/ID-CRC-176) filed 
December 22, 1997, the entire contents of which are herein incorporated by reference for all 
25 purposes. 

[73] The user concepts information may be configured by the user or may 
be altematively be automatically compiled by server system 1 12 by monitoring the user's 
interactions with the user system and by monitoring information accessed by the user. 
Several tools may be provided to configure the contents of the user profile. One such user 
30 profile editor is described in U.S. Patent Application Serial No. 08/995,616. 

[74] Referring back to Fig. 6, for each UDIO, server system 112 then 
determines a degree of relevancy of the UDIO to each concept of interest to the user 
determined in step 602 (step 604). According to an embodiment of the present invention, the 
output of step 604 may be represented by a table such as Table 1 shown below: 
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TABLE 1: UDIO relevance table 



UDIO 


Concept of 
interest to 
the User 


Degree of 
relevancy of the 
UDIO to the 
concept of interest 
to the user 


UDIOi 


c, 


Rii 


UDIOi 


C2 


R12 








UDIOi 


Cm 


Rim 


UDIO2 


c, 


R21 


UDIO2 


C2 


R22 








UDIO2 


Cm 


R2m 














UDIOn 


c, 


Rnl 


UDIOn 


C2 


Rn2 








UDIOn 


Cm 


Rnm 



where: 



"n" 


Indicates the total number of extracted UDIOs 


"m" 


Indicates the total number of concepts of interest to the user 


"UDIOi" 


Indicates the i* UDIO 


"Ci" 


Indicates the j user concept of interest 


"Rij" 


Denotes a value indicating the degree of relevancy of UDIOi to user 
concept of interest Cj (e.g. R12 indicates the degree of relevancy of 
UDIOi to user concept C2). The degree of relevancy value may be 
zero if the UDIO is not relevant to the concept. 



[75] Several different techniques may be used by server system 1 12 to 
determine the degree of relevancy. According to one technique, server 112 may compare the 
content information associated with a UDIO with the keywords and their associated 
probability values {''<Keywords_and _probabilities> associated with the concept and 
determine the degree of relevancy based upon the comparison. Other techniques to determine 
the degrees of relevancy have been described in U.S. Patent Applications 08/995,616, 
09/636,039, 09/483,092, and 09/483,094, the entire contents of which have been incorporated 
by reference for all purposes. Several other techniques known to those of skill in the art may 
also be used to determine the degrees of relevancy. 
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[76] Server system 112 may build a table similar to Table 1 for CPIOs. The 
table built for CPIOs (as shown in Table 2) may store degree of relevancy information for the 
CPIOs as they relate to each concept of interest to the user. Server system 112 may build the 
CPIO relevance table after contents for the CPIOs have been determined using one or more 
content recognition techniques. 

TABLE 2: CPIO relevance table 





interest to 
the User 


relevancv of the 
CPIO to the 
concept of interest 
to the user 


CPIOi 


c, 


Rii 


CPIOi 


C2 


R12 








CPIOi 


Cm 


Rim 


CPI02 


c, 


R2I 


CPI02 


C2 


R22 








CPI02 


C]T1 


R2m 














CPIOb 


c, 


Rpi 


CPIOo 


C2 


Rp2 








CPIOp 


Cm 


Rpm 



where: 



"p" 


Indicates the total number of CPIOs 


"m" 


Indicates the total number of concepts of interest to the user 


"CPIOk" 


Indicates the k^*" CPIO 


"Cj" 


Indicates the user concept of interest 




Denotes a value indicating the degree of relevancy of CPIOk to user 
concept of interest Cj (e.g. R12 indicates the degree of relevancy of 
CPIOi to user concept C2). The degree of relevancy value may be 
zero if the CPIO is not relevant to the concept. 



[77] Based upon the information contained in the UDIO relevance table 
(e.g. Table 1) and the CPIO relevance table (e.g. Table 2), for each CPIO, server system 112 
determines the degree of relevancy (or a relevance score) of the CPIO to each UDIO as it 
relates to a particular user concept of interest (i.e. the server determines the degree of 
relevancy of a CPIO to a (UDIO, Concept) pair) (step 606). According to an embodiment of 
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the present invention, the degree of relevancy of a CPIO may be calculated for only those 
UDIOs whose degree of relevancy ("R" shown in Table 1) to a particular user concept of 
interest is above a user-configurable threshold value. This helps reduce the amount of 
processing performed by server 112. 
5 [78] According to an embodiment of the present invention, the output of 



step 606 may be represented by Table 3 shown below: 

TABLE 3 



UDIO 


Concept 

of 
interest 
to the 

User 


Degree of 
relevancy of 
the UDIO to 
the concept of 
interest to the 
user 


List of CPIOs and their relevancy to a UDIO as it 
applies to a particular concept of interest to the user 


UDIOi 


c, 


Rii 


<(CPIO,, RRiu), (CPIO2, RR2n), • • . (CPIOo, RRoii)> 


UDIOi 


C2 


R12 


<(CPIOi, RRi,2), (CPIO2, RR212), . . . (CPIOo, RRoi2)> 










UDIO, 


Cm 


Rim 


<(CPIOi, RRllm), (CPIO2, RR21m), . . . (CPIO„, RR„ini)> 


UDIO2 


c, 


R2I 


<(CPIO,, RR,2i), (CPIO2, RR221), . . . (CPIOo, RRo2i)> 


UDIO2 


C2 


R22 


<(CPIO,, RR,22), (CPIO2, RR222), . . . (CPIOo, RRo22)> 










UDIO2 


Cm 


R2m 


<(CPIOl, RRl2m), (CPIO2, RR22m), • • • (CPIOo, RR„2m)> 


















UDIOn 


c, 


Rnl 


<(CPIO,, RR,n,), (CPIO2, RR2nl), . . . (CPIO„, KR^nl)> 


UDIOn 


C2 


Rn2 


<(CPIO,, RR,„2), (CPIO2, RR2n2), . . . (CPIO„, RRon2)> 










UDIOn 


Cm 


Rnm 


<(CPIOi, RRlnm), (CPIO2, RR2nm), • • • (CPIOp, RRonm)> 



where: 



"n" 


Indicates the total number of extracted UDIOs 


"m" 


Indicates the total number of concepts of interest to the user 


"P" 


Indicates the total number of CPIOs 


"UDIOi" 


Indicates the i'*" UDIO 


"Ci" 


Indicates the j^*^ user concept of interest 


"CPIOk" 


Indicates the k'*" CPIO 




Denotes a value indicating the degree of relevancy of UDIOi to user 
concept of interest Cj (e.g. R12 indicates the degree of relevancy of 
UDIOi to user concept C2). The degree of relevancy value may be 
zero if the UDIO is not relevant to the concept. 


RRkij 


Denotes a value indicating the degree of relevancy of CPIOk to 
UDIOi as it relates to user concept of interest Cj. This value may be 
zero if a particular CPIO is not relevant to a particular UDIO as it 
relates to a particular user concept of interest. According to an 
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embodiment of the present invention, value RRkij may be based upon 
the value of Rij in Table 1 (i.e. the degree of relevancy of UDIOi to 
user concept of interest Q) and the value of Rkj in Table 2 (i.e. the 
degree of relevancy of CPIOk to a particular user concept of interest 
[Cii 

[79] The list of CPIOs for each UDIO as it relates to a particular user 
concept of interest may be ranked or ordered based upon the degrees of relevancy of the 
CPIOs (i.e. based on the value of RR). Several techniques may be used by server 1 12 to 
5 determine the relevancy of CPIOs to a particular UDIO as it relates to a particular user 

concept of interest. According to a one technique, a simple linear combination may be used. 
According to the linear combination technique, the value of RR may be expressed as follows: 

RRkij = (Wi * Rij) * (W2 * Rkj) 



where 



RRkij 


Denotes a value indicating the degree of relevancy of CPIOk to 
UDIOi as it relates to user concept of interest Ci. 


"Rij" 


Denotes a value indicating the degree of relevancy of UDIOi to user 
concept of interest Ci. 


"Rkj" 


Denotes a value indicating the degree of relevancy of CPIOk to user 
concept of interest Ci. 


Wi 


Is a real-valued weight that expresses the relative importance 
assigned to the relevance value of a UDIO 


W2 


Is a real-valued weight that expresses the relative importance 
assigned to the relevance value of a CPIO 



10 

[80] Several other techniques known to those of skill in the art may also be 
used to determine the degrees of relevancy. A few such techniques have been described in 
U.S. Patent Apphcations 08/995,616, 09/636,039, 09/483,092, and 09/483,094, the entire 
contents of which have been incorporated by reference for all purposes. 

15 [81] According to an embodiment of the present invention, the list of CPIOs 

along with their relevancy information may then be used for further processing according to 
step 616 described below wherein the results generated by the various selection techniques 
are combined to generate a single "final" list of one or more CPIO objects which is then 
communicated to the user system used by the user. 

20 [82] According to an alternative embodiment of the present invention, 

instead of commimicating a single CPIO list, the lists generated by the various selection 
techniques may be communicated to the user system. In this embodiment, for each selection 
technique, server system 112 selects one or more CPIOs based on the relevancy information 
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generated for the CPIOs using the particular selection technique (step 608). Various 
techniques may be used by server 1 12 to perform the selection of CPIOs. According to one 
technique, the selection is based on the degree of relevancy of the CPIOs to each UDIO as it 
relates to a particular user concept of interest, i.e. the selection is based on the value of "RR" 
shown in Table 3. Server system 112 may be configured to select only those CPIOs w^hose 
"RR" value is above a user-configurable threshold value. According to another technique, 
server system may be configured to select CPIOs based on the "RR" value and also based on 
the degree of relevancy of the UDIOs to the user concepts of interest (i.e. based on the value 
of "R" shown in Table 1 and 3). Other techniques may use other criteria for selection of the 
CPIOs. It should be apparent that various other selection techniques known to those skilled 
in the art may also be used in accordance with the present invention. The CPIOs selected in 
step 608 may then be communicated to user system 102 according to step 312 in Fig. 3. 

[83] As previously indicated, server 112 may also select CPIOs using a 
"class-based" or "type-based" selection technique as denoted by steps 610 and 612 in Fig. 6. 
As shown in Fig, 6, the class-based selection technique is initiated when server system 112 
determines one or more comparison techniques to be applied to each UDIO based upon the 
class of the UDIO (step 610). As described above, the class or type of an information object 
indicates the type of content or information stored by the information object, and is usually 
determined when the information object is extracted (or generated). For example, class 
information for the UDIOs is determined in step 402 of Fig. 4. 

[84] In general, a comparison technique enables comparison between the 
content stored by the information objects. As previously described, various content 
recognition techniques may be applied to determine the content information of each 
information object. For example, according to an embodiment of the present invention, the 
content information for an UDIO is determined in step 406 of Fig. 4. One or more 
comparison techniques may be provided for each information object class. Accordingly, the 
comparison techniques determined by server 1 12 in step 610 to be applied to an UDIO 
depend on the type of the UDIO. Using a comparison technique, the server system compares 
contents of an UDIO with contents of a CPIO to determine the relevancy (a degree of 
relevancy or a relevance score) of the CPIO to the UDIO. Information related to the 
comparison techniques for the various classes may be stored in database 110. 

[85] Table 4 shown below provides examples of classes and comparison 
techniques corresponding to the classes according to an embodiment of the present invention. 
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TABLE 4: Comparison techniques for the various classes 



Class 


Corresponding Comparison Techniques 


Text 


Text comnarijson techninues 


Image 


(1) Text comparison techniques 

(2) Techniques which determine similarity between images 


Audio 


(1) Text comparison techniques which compare transcript(s) 
of audio information objects 

(2) Sound source comparison techniques 

(3) Sound similarity comparison techniques 


Video 


(1) Text comparison techniques 

(2.^ Techniflues which determine similaritv between 

J A, \^\^XMJL±.M.\A\A-\^\J tT XXX VXX \i* W LWX XXXXXXW OXXXXXXCtX X k V L^WkVVWvXX 

keyframes 

(3) Text comparison techniques which compare transcript(s) 
of audio associated with the video information object 

(4) Sound source comparison techniques 

(5) Sound similarity comparison techniques 

(6) Video similarity comparison techniques 



Ly [86] As shown in Table 4, for a text class object, text comparison 

7 techniques may be used which compare the text associated with a text information object 

5 with text associated with other information objects. For example, for a text UDIO, text 
H comparison techniques may compare the text associated with the UDIO with text associated 
g with a CPIO. 

^ [87] For an image object, text comparison techniques, image similarity 

techniques, and other techniques known to those skilled in the art may be used. A text 

10 comparison technique compares the text associated with an image object with text associated 
with other information objects. For example, for an image UDIO, text comparison 
techniques may compare the text associated with the image UDIO with text associated with a 
CPIO. Image similarity techniques determine similarity between images associated with the 
information objects. For example, for an image UDIO, an image similarity technique 

15 compares the image associated with the image UDIO with images associated with CPIOs. 
Virage, Inc. of San Mateo, Califomia, provides an image similarity application that may be 
used to compare images. 

[88] For an audio information object, text comparison techniques, sound 
source comparison techniques, sound similarity techniques, and other techniques known to 

20 those skilled in the art may be used. Text comparison techniques compare the text transcripts 
associated with an audio information object with text associated with other information 
objects. For example, for an audio UDIO, text comparison techniques may compare the text 
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transcript associated with the audio UDIO with text associated with a CPIO. Soiind source 
comparison techniques and sound similarity techniques compare sources of the audio signals 
stored by an audio information object. For example, a sound source comparison technique 
may compare the audio signals of an audio UDIO with the audio signals of an audio CPIO to 
5 determine the degree of similarity of the audio sources. 

[89] All of the comparison techniques associated with a text information 
object, an image information object, and an audio information object, along with video 
comparison techniques may be applied to compare video information objects. For example, 
text comparison techniques may be used to compare text transcripts associated with a video 
10 information object, image comparison techniques may be used to compare images or 
Q keyframes associated with a video information object (e.g. applications provided by Virage, 
^ Inc. which compare similarity between video frames), and audio comparison techniques may 
O be used to compare audio signals associated with the video information object. Video 
Q comparison techniques may also be used to compare video signals associated with the video 
y 5 object. The above-mentioned comparison techniques for a video information object may be 
5 used to compare a video UDIO and other CPIOs. 

n [90] It should be apparent that the scope of the present invention is not 

^ limited to the techniques shown in the Table 4. Various other comparison techniques known 
Q to those skilled in the art may also be used to compare the contents of information objects 
^0 (including various other types of information objects not shown in Table 4). 

[91] Referring back to Fig. 6, for each UDIO, each comparison technique 
associated with the type of the UDIO and determined in step 610 is applied to determine 
relevancy scores between the UDIO and the CPIOs (step 612). Accordingly, for a particular 
UDIO, the output of step 612 may comprise a list of CPIOs for each comparison technique 
25 and a relevancy score indicating the relevancy of each CPIO and the particular UDIO for a 
particular matching technique. According to an embodiment of the present invention, the 
output of step 612 may be stored in a table such as Table 5 shown below: 
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TABLES 



UDIO 


Class 


Comparison 


List of relevant CPIOs and relevancy 






Techniques 


scores 




UDIOi 


Class X 


Technique #1 


<(CPIOi, RRRiii), (CPIO2, RRR211), . 








Technique #2 


<(CPIOi, RRR121), (CPIO2, RRR221), . 


. > 


UDIO2 


Class Y 


Technique #1 


<(CPIOi, RRR112), (CPIO2, RRR212), . 


. > 






Technique #4 


<(CPIOu RRR142), (CPIO2, RRR242), . 


. > 






Technique #5 


<(CPIOi, RRR152), (CPIO2, RRR252), . 


. > 










UDIOn 


Class X 


Technique #1 


<(CPIOi, RRRlln), (CPIO2, RRR21n), . 


. > 






Technique #2 


<(CPIOl, RRRl2n), (CPIO2, RRR22n), . 


. > 



where: 

"RRRxyz" - Indicates a degree of relevancy of CPIOx to UDIOz determined by 



Q applying comparison technique "y" (Technique #y) . 

^ 5 [92] In the embodiment shown above, Table 5 comprises four columns. 

O The first column identifies a UDIO (e.g. UDIOi indicates the first user document information 
Q object). The second column indicates the class or type of the UDIO identified in first column 
y (e.g. UDIOi is of class "X," UDIO2 is of class "Y," and UDIOn is also of class "X."). The 
s third column identifies the comparison techniques corresponding to the class identified in the 

CJO second column (e.g. two comparison techniques are associated with class "X", namely 
^ technique #1 and technique #2; three comparison techniques apply to class "Y", namely 
O techniques #1, #4, and #5, and so on). For each comparison technique identified in the third 
~ column, the fourth column contains a list of CPIOs and relevancy scores of the CPIOs to the 
UDIO object (identified in the first column) determined by applying the comparison 
15 technique identified in the third column. 

[93] Several different techniques may be used by the present invention to 
determine the RRR scores. According to a specific technique, for each UDIO of a given 
class, server 112 applies a comparison technique associated with the class of the UDIO to 
each CPIO which has a similar comparison technique associated with it. For example, for a 
20 text UDIO, a text comparison technique associated with the text class can be applied to image 
CPIOs, audio CPIOs, video CPIOs, and CPIOs of other types which have a text comparison 
technique associated with them to compare the text transcripts associated with the 
information objects. An UDIO of a first class and a CPIO of a second class (which may or 
may not be the same class as the first class) are said to be compatible for a particular 
25 comparison technique if that particular comparison technique is associated with the first and 
the second class. Accordingly, if a UDIO is a text object then it is compatible with CPIOs of 
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classes that have a corresponding text comparison technique. For example, if a UDIO is of 
type image, and an image -similarity comparison technique is being used, then image CPIOs 
and video CPIOs are compatible since there is a corresponding image similarity comparison 
technique associated with the image and video classes. For an audio UDIO, if a sound source 
5 comparison technique is being used, then CPIOs that are either audio or video objects are 
compatible since there is an corresponding sound source comparison technique associated 
with each of those classes. The compatibility of a UDIO and CPIO for a given comparison 
technique can be determined by consulting the information stored in Table 4. 



10 generated by applying the particular comparison technique is associated with the CPIO for 
^ the particular UDIO and the particular comparison technique and stored in memory (e.g. in 
^ Table 5). For example, server 112 may use a sound source comparison technique to compare 
O sound sources in an audio UDIO with sound sources in audio or video CPIOs. Using the 
SI: sound source comparison technique, server 112 determines unique sound sources (e.g. 
W5 speakers) by analyzing passages of speech data in the audio UDIO and compares each sound 
5 source to sound sources determined in the compatible CPIOs. For a particular compatible 
CPIO, when server 112 determines that a sound source in the UDIO occurs in the CPIO, a 
H= confidence score associated with the particular CPIO is incremented. After all the passages 
p in the particular UDIO have been compared in this way to sound sources in the particular 
^^0 CPIO, the average confidence score for the particular CPIO is calculated by dividing the 
aggregated confidence score for the CPIO by the number of speakers identified in the 
particular UDIO. The average confidence score is output as the relevance RRR value of the 
particular CPIO to the UDIO for the sound source comparison technique. 



[94] For two compatible information objects, the relevance score RRR 



[95] According to an embodiment of the present invention, the above- 
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described procedure may be described by the following pseudo-code: 
UDIOy_Speakers = Speakers identified in audio UDIOy; 
#UDIOy_Speakers = Number of Speakers in UDIOy Speakers; 
CPIOx_Speakers = Speaker identified in compatible CPIOx; 
confidence_score = 0; //Initialize to zero 
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for each Speaker in UDIOy_Speakers { 



By applying Technique#z determine if Speaker is included in CPIOx_Speakers; 
if Speaker is included in CPIOx_Speakers { 
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} 

RRRxyz = confidence_score/#UDIOy_Speakers; 

[96] If the UDIO and CPIO are not compatible for a given comparison 
5 technique (e.g. a text CPIO is not compatible with a video UDIO for a video comparison 
technique), the value of RRR associated with the CPIO may be set to zero. 

[97] Several other techniques known to those of skill in the art may also be 
used to determine the degrees of relevancy. Examples of other methods for determining the 
relevancy scores have been described in U.S. Patent Applications 08/995,616, 09/636,039, 
10 09/483,092, and 09/483,094, the entire contents of which have been incorporated by 
O reference for all purposes. 

5 [98] After all the UDIO objects have been processed, according to an 

embodiment of the present invention, the list of CPIOs along with their relevancy information 

□ may then be used for further processing according to step 616 described below wherein the 

S5 results generated by the various selection techniques are combined to generate a single 

"final" list of one or more CPIOs which is then communicated to the user system used by the 

^4 user. 

h I [99] According to an altemative embodiment of the present invention, 

^ instead of communicating a single CPIO list, the results of the various selection techniques 

20 may be communicated to the user system. In this embodiment, server system 112 selects one 
or more CPIOs based on the relevancy information generated for the CPIOs (step 614). 
Various techniques may be used by server 1 12 to select the CPIOs. According to one 
technique, the selection is based on the degree of relevancy of the CPIOs to each UDIO for a 
particular comparison technique, i.e. the selection is based on the value of "RRR" shown in 

25 Table 5. Server system 112 may be configured to select only those CPIOs whose "RRR" 
value is above a user-configurable threshold value. Other techniques may use other criteria 
for selection of the CPIOs. It should be apparent that various other selection techniques may 
also be used in accordance with the present invention. The CPIOs selected in step 614 may 
then be communicated to user system 102 according to step 312 in Fig. 3. 

30 [100] As described above, according to an embodiment of the present 

invention, the results of one or more selection techniques (such as class-based selection 
techniques and concept-based selection techniques) are combined to form a single "final" list 
of CPIOs (step 616) which is then communicated to user system 102 according to step 312 in 
Fig. 3. According to the teachings of the present invention, various combination techniques 
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may be used by server 1 12 to combine the various CPIOs into one "final" list. Each 
combination technique takes as input CPIOs and their associated relevancy information 
(relevancy scores) generated by applying the various selection techniques, and outputs a 
single list of, possibly ranked, one or more CPIOs. The CPIOs included in the single "final" 
5 list of CPIOs may be selected based upon criteria such as concepts of interest to the user, the 
context in which the user is viewing the user document, contents of the document (or sections 
thereof) viewed by the user, contents of the CPIOs, relevancy information, and the like. By 
appropriately configuring the combination technique, the class(es) of information objects 
may also be used as criteria for generating the final combined list of CPIOs. The CPIOs 
10 included in the final single combined list of CPIOs are then communicated to user system 
102 according to step 312 in Fig. 3. 

[101] According to an embodiment of the present invention, a "source- 

□ based" combination technique is used to combine the results of the various selection 

Ln 

techniques to generate a single list of, possibly ranked, CPIOs. Fig. 7 is a simplified block 
^5 diagram showing the inputs and outputs of a source-based combination technique according 
= to an embodiment of the present invention. As shown in Fig. 7, the source-based 
^ combination technique 702 takes as input CPIOs and their associated relevancy information 
H (scores) generated by one or more "sources". Input 704 to combination technique 702 (as 
p shown in Fig. 7) may be represented as follows: 
^0 Source#l ListElement#l , ListElement#2, . . . 

Source#2 ListElement#l , ListElement#2, . . . 
Source#3 ListElement#l , ListElement#2, . . . 
and so on ,wherein each ListElement comprises a CPIO and a value indicating the degree of 
relevance or relevance score of the CPIO to the source. For example, each list element may 
25 be represented as 

ListElementb = (CPIOa, Relab) 
where, 

ListElementb — indicates a ListElement belonging to a list corresponding to 
Source#b. 

30 CPIOa - identifies a CPIO; and 

Relab — indicates the relevance of CPIOa to Source#b. 
The combination technique 702 outputs a single (possibly ordered) list 708 of CPIOs based 
on user-configurable parameters 706 associated with the combination technique. 
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[102] According to an embodiment of the present invention, each "source" 
identifies a UDIO and information related to a selection technique applied to the UDIO. 
Accordingly, a source may be represented as shown below according to an embodiment of 
the present invention: 
5 Source = (UDIO, SelectionMethodlnfo) 

where: 

UDIO - Identifies a UDIO; and 

SelectionMethodlnfo - Identifies information about a selection technique 
(or a particular aspect of a selection technique) applied to the 
1 0 UDIO to generate the relevancy information for one or more 

C3 CPIOs associated with the source. 

iQ [103] For a concept-based selection technique described above, the 

rz SelectionMethodlnfo field may identify a user concept. Accordingly, for a concept-based 
P selection method, a source may be identified by a combination of an UDIO and a concept of 
^1 5 interest to the user as follows: 
L, Source = (UDIO, UserConcept) 

H [104] Examples of sources for a concept-based selection technique may 

include (UDIO#l, UserConcept="Travel in Paris"), (UDIO#l, UserConcept="Footbair'), 
(UDIO#2, "Presidential Elections"), and so on. For example, the first two columns of Table 
20 3 in conjunction identify a source, and column four of Table 3 identifies CPIOs and 
associated relevancy scores included in the list elements associated with the source. 

[105] For a class-based selection technique, the SelectionMethodlnfo field 
may identify the class of a particular UDIO and a particular comparison technique applied to 
the UDIO to generate CPIOs and their relevancy scores for the particular class of UDIO 
25 using the particular comparison technique. Accordingly, for a class-based selection method, 
a source may be represented by: 

Source = (UDIO, (Class_of_M)/(9, Comparison Technique)), 
[106] Examples of sources generated by a class-based selection technique 
may include (UDIO#l, (Class="Text Object", ComparisonTechnique- 'Text comparison")), 
30 (UDIO#2, (Class-'Audio Object", ComparisonTechnique="Sound source comparison")), 
(UDIO#2, (Class="Audio Object", ComparisonTechnique="Text comparison")), (UDIO#2, 
(Class="Video Object", ComparisonTechnique- 'Video comparison")), and the like. For 
example, the first three columns of Table 5 in conjunction identify a source, and colimin four 
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of Table 3 identifies CPIOs £ind associated relevancy scores included in the list elements 
associated with the source. 

[107] Accordingly, for a source (UDIO, UserConcept) (i.e. a source for a 
concept-based selection technique for a particular UDIO as it relates to a particular user 
concept of interest), the Relab value for each ListElement associated with the source indicates 
the relevance of a CPIOa to the UDIO and user concept indicated by the source (i.e. Relab 
corresponds to the value of "RR"). For a source (UDIO, (Class of C/D/O, Comparison 
Technique)) (i.e. a source for a class-based selection technique for a particular UDIO of a 
particular class and a comparison technique associated with the class), the Relab value for 
each ListElement indicates the relevance of a CPIOa to the UDIO of the type generated using 
the particular comparison technique (i.e. Relab corresponds to the value of "RRR" depicted in 
Table 5). 

[108] As shown in Fig. 7, one or more user-configurable parameters 706 may 
be input to the combination technique to facilitate the combination process. These 
parameters 706 may include one or more "weighting factors" 706-a, thresholds 706-b, etc. A 
"weighting factor" may be associated with combination criterion, e.g., a weighing factor may 
be associated with a particular source to specify the importance of the particular source. For 
example, if the user would like to increase the chances of a CPIO for a particular source 
being selected in the "final" list, the user may associate a high weighting factor value with the 
particular source to emphasize the significance of the source. A threshold relevance value 
may also be associated with combination criteria, e.g., a threshold may be associated with a 
particular source to indicate a minimum degree of relevancy value for a CPIO to be selected 
in the final single list. For example, a user may associate a first threshold with a first source, 
and as a result, only those CPIOs whose relevancy scores are above the first threshold and 
which are included in the list elements for the particular source will be selected for inclusion 
in the single "final" list. 

[109] Accordingly, weighting factors 706-a and thresholds 706-b allow a 
user to customize the manner in which combination technique 702 will select and possibly 
rank CPIOs in the final list of CPIOs. For example, by setting appropriate weighting factors 
and thresholds, the user may emphasize the significance of a particular source, emphasize the 
significance of a particular concept of interest to the user, emphasize the significance of a 
particular type of information object, and the like. 

[110] Fig. 8 is a simplified high-level flowchart showing processing 
performed during step 616 of Fig. 6 for generating a single "final" list of CPIOs according to 
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an embodiment of the present invention. The flowchart depicted in Fig. 8 is merely 
illustrative of an embodiment incorporating the present invention and does not limit the scope 
of the invention as recited in the claims. One of ordinary skill in the art would recognize 
other variations, modifications, and altematives. According to an embodiment of the present 
5 invention, the method depicted in Fig. 8 is performed by modules executing on server system 
112. These modules may include software modules, hardware modules, or combinations 
thereof. 

[Ill] As shown in Fig. 8, the method is initiated by initializing various 
variables that will be used for the computation (step 802). According to an embodiment of 
10 the present invention, these variables may include (a) an array "NumVotes" comprising "N" 
5 elements, where "N" is the total number of CPIOs, wherein each element of the array 
^ corresponds to a particular CPIO; (b) an array "Weight Votes" comprising "N" elements, 
Lfl wherein each element of the array corresponds to a particular CPIO; (c) an array of lists 
2 "SourcesVoting" comprising "N" elements where each element of the array points to a list of 
^5 CPIOs; (d) a variable "#S' which is initialized to the total number of sources to be processed; 
□ and (e) a counter variable "d" which is initialized to one and is used to traverse the sources. 
^ : [112] The "Num Votes" array is used to record the number of times that a 

particular CPIO was considered to be included in the "final" single list of CPIOs. 
M= Accordingly, an element of array "Num Votes" corresponding to a particular CPIO indicates 
20 the number of times that the particular CPIO was considered by the combination algorithm to 
be included in the "final" single list of CPIOs. Each element of the "WeightVotes" array 
corresponding to a particular CPIO accumulates the "weighting factor" as applied to the 
source for the particular CPIO. Each element of the "SourcesVoting" array corresponding to 
a particular CPIO points to a list which indicates the sources which contributed to that 
25 particular CPIO. 

[113] After the various variables have been initialized, processing of the 
sources is then commenced. Server 112 determines if there are any unprocessed sources 
(step 804). This may be accomplished by checking if the value of the "d" variable is greater 
than the "#S" variable value. If the value of the "d" variable is greater than the value of the 
30 "#S" variable, it indicates that all the sources have been processed and processing continues 
with step 822. However, if the value of the "d" variable is not greater than the "#S" variable, 
it indicates the existence of unprocessed sources and processing continues with step 806. 
Accordingly, the "d" variable is used to traverse through the available sources and process 
them one at a time. 
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[114] In step 806, the present invention selects a particular source (Sd 
indicating the d^*^ source) for processing. Variables used for processing the list elements for 
the source Sd selected in step 806 are then initialized (step 808). According to an 
embodiment of the present invention, these variables may include (a) a variable "#L" which 
is initiaUzed to the number of elements {ListElements) in the list corresponding to source Sd; 
and (b) a counter variable "e" which is initialized to one and which is used to traverse the 
elements in the list corresponding to source Sd- 

[115] Server system 1 12 then determines if there are any unprocessed 
elements in the list of elements corresponding to the source (Sd) selected in step 806 (step 
810). This may be accomplished by checking if the value of the "e" variable is greater than 
the "#L" variable value. If the value of the "e" variable is greater than the value of the "#L" 
variable, it indicates that all the list elements have been processed and processing continues 
with step 820 wherein the value of the "d" variable is incremented by one to select the next 
source. However, if the value of the "e" variable is not greater than the "#L" variable, it 
indicates the existence of unprocessed list elements and processing continues with step 812. 
Accordingly, the "e" variable is used to traverse through the list elements for a particular 
source and process the list elements one at a time. 

[116] In step 812, the "CPIOde" variable is initiaUzed to point to the CPIO 
referenced by the e^*^ list element in the list for source Sd, and the "Rde" variable is initialized 
to store or refer to the relevance value or score associated with CPIO referenced by the e*^ list 
element in the list for source Sd. The value of R<ie is then checked to determine if it is greater 
than a user-configurable value Tsd (step 814), where Tsd indicates a threshold value 
configured by the user for source Sd. If the value of Rde is not greater that the value of Tsd, 
the e^*^ list element is not considered for inclusion in the "final" list of CPIOs (i.e. the list 
element is skipped) and processing continues with step 818 wherein the value of "e" is 
incremented by one to select the next list element from the list corresponding to source Sd. If 
the value of R<ie is greater that the value of Tsd? it indicates that the particular CPIO included 
in the e**^ hst element is to be considered for inclusion in the "final" single list and processing 
continues with step 816. 

[117] In step 816, the value of the element of array "NumVotes" 
corresponding to the CPIO indicated by variable CPIOde is incremented by one. The value of 
the element of array "Weight Votes" corresponding to the CPIO indicated by variable CPIOde 
is incremented by the user-configurable weighting factor (Wsd) associated with source Sd. 
Information identifying source Sd is also appended to the list pointed to by the element of 
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array "Sources Voting" corresponding to the CPIO indicated by variable CPIOde- Processing 
then continues with step 818 wherein the value of "e" is incremented by one to select the next 
list element from the list corresponding to source Sd. 

[118] After all the sources and their corresponding list elements have been 
5 processed (as determined by step 804), a "final" single list of CPIOs is generated based on 
user-specified criteria and information recorded in the "Num Votes," "WeightVotes," and 
"Sources Voting" arrays (step 822). According to an embodiment of the present invention, 
the user-specified criteria may configure the present invention to select CPIOs to be included 
in the "final" list based on information stored in the "WeightVotes" array. For example, only 
10 those CPIOs having WeightVotes values above a user-configurable threshold may be selected 
for inclusion in the "final" list. Alternatively, the "WeightVotes" array may be sorted and 
□ CPIOs corresponding to the top "G" (where the value of "G" is user-configurable) 
,n WeightVotes values may be selected for inclusion in the "final" single list of CPIOs to be 

communicated to user system 102. In other embodiment, server system 112 may be 
fflS configured to select CPIOs based on information contained in the "Num Votes" array. For 
S example, in this embodiment, only those CPIOs having Num Votes values above a user- 
^ configurable threshold are selected for inclusion in the "final" list of CPIOs. Altematively, 
s] the "Num Votes" array may be sorted and CPIOs corresponding to the top "H" (where the 
value of "H" is user-configurable) Num Votes values may be selected for inclusion in the 
3© "final" list to be communicated to user system 102. 

[119] CPIOs may also be selected based upon the sources that contributed 
the CPIOs. For example, the present invention may be configured to select a CPIO based 
upon information contained in the lists pointed to by elements of the "SourcesVoting" array. 
The user may also specify selection criteria that is based on a combination of information 
25 stored in the Num Votes, WeightVotes, and/or SourcesVoting arrays. It should be apparent 
that the scope of the present invention is not restricted to the selection criteria/techniques 
discussed above. Various other techniques known to those of ordinary skill in the art may 
also be used to select CPIOs to be included in the final single list of CPIOs to be 
communicated to the user system. 
30 [120] The "final" single list of one or more CPIOs generated in step 822 may 

then be communicated to user system 102 according to step 312 in Fig. 3. The information 
stored by the CPIOs included in the "final" list may then be output to the user by user system 
102 (step 314 in Fig. 3). 
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[121] Figs. 9A, 9B, and 9C depict examples of user interfaces which may be 
used to present or output information stored by the selected CPIOs to the user according to 
embodiments of the present invention. The user interfaces depicted in Figs. 9A, 9B, and 9C 
are mere examples and should not be construed to limit the scope of the claims herein. One 
5 of ordinary skill in the art would recognize many other variations, alternatives, and 
modifications. 

[122] Fig. 9 A depicts a browser screen 900 having a current document 
display area 902 and a selected information display area 904. The document viewed by the 
user is displayed in current document display area 902. For example, in Fig. 9A, the user is 
10 viewing a document entitled, "The Design of a Wearable Computer." 
p [123] Selected information display area 904 is used to display information 

stored by CPIOs selected by server system 112 and cormnunicated to the user system in 
□ accordance with the teachings of the present invention. The information displayed in selected 
Q information display area 904 may include text information, image information, audio 
^5 information, video information, URL information, and other types of information and 
s combinations thereof. The information displayed in selected information display area 904 
may have been selected by server 112 using one or more of the selection techniques 

^ described above. For example, in Fig. 9A, the information displayed in area 904 includes 

ry 

p text and URL links that are determined by server 1 12 to be relevant in some way to the 
'^0 document viewed by the user and to concepts of interest to the user. The URL links includes 
links to documents entitled "Cyborg Seeks. . "Tourist Wear. . and "Smart Clothing. . 
which may be of interest to the user. In this specific embodiment, the document names have 
been shortened to accommodate the size of information display area 904. However, the 
content and the display arrangement of the information may vary in alternative embodiments 
25 of the present invention. Further, the number of selected information display areas may also 
vary in alternative embodiments of the present invention. 

[124] In the embodiment depicted in Fig. 9 A, area 906 displays concepts of 
interest to the user of the user system. As described above, this information may be stored in 
a user profile. In the example depicted in Fig. 9A, the user concepts of interest include 
30 concepts identified by content identifiers "Agents," "Interfaces," "Wearable," "Bayes," 

"Design," and "InputDev". For each concept, the user interface also indicates the degree of 
relevancy of the document viewed by the user in area 902 to the concepts of interest. For 
example, the document viewed by the user is 97% relevant to the concept identified by 
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concept identifier "Wearable". It should be apparent the user concepts need not be displayed 
in alternative embodiments of the present invention. 

[125] The user may make selections from the information displayed in 
selected information display area 904. For example, selection of URL link "Cyborg Seeks . . 
5 in Fig. 9A causes the web page entitled "Cyborg Seeks Community" corresponding to the 
selected URL to be loaded in area 902 of browser 900 as depicted in Fig. 9B. The 
information displayed in selected information display area is also dynamically updated to 
display information that is relevant to the "new" document displayed in area 902. In this 
embodiment, a change in the document viewed by a user causes server 1 12 to select CPIOs 
10 based upon the "new" document. Selected information display area is updated to display 
information stored in CPIOs selected by server system 112 and communicated to the user 
system. Accordingly, as shown in Fig, 9B, new information is displayed in selected 

0 information display area 904. Concept indicators displayed in are 906 are also updated to 

2 provide indications of relevance of the current document, "Cyborg Seeks Community," to the 
'^15 plurality of concepts that are of interest to the user. 

B [126] Fig. 9C depicts yet another user interface according to another 

rj embodiment of the present invention. Fig. 9C depicts a browser screen 910 having a current 
document display area 912 and a selected information display area 914. The document 

1 y 

□ viewed by the user is displayed in current document display area 902. For example, in Fig. 
^20 9C, the user is viewing a document entitled, "Digital Manipulatives: New Toys To Think 
With." 

[127] In the embodiment depicted in Fig. 9C, area 916 displays a plurality of 
channels of interest to the user. Information related to the channels may be stored in a user 
profile. In the embodiment depicted in Fig. 9C, the channels include a shopping channel, a 

25 news channel, a money channel, and others. Different channels can be specified in other 

embodiments. Selected information display area 914 is used to display information stored by 
CPIOs selected by server system 1 12 in accordance with the teachings of the present 
invention based upon the document viewed by the user in area 912 and based upon channels 
of interest to the user displayed in area 916. For example, in Fig. 9C, the information 

30 displayed in area 904 includes information that is relevant in some way to the document 
viewed by the user and channels of interest to the user and of possible interest to the user. 

[128] As described above, the present invention provides techniques for 
automatically providing information to users based upon information or documents accessed 
or viewed by the user. Since the present invention does not require specific user input to find 
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the relevant information, the invention can be easily used to users who are not computer 
literate or are uncomfortable using computers. Further, the present invention allows content 
providers to target information to users who are likely to be Interested in the information. 
The present invention can thus be used as a powerful advertising tool for targeting 
advertisement information to users. The present invention may also be used for various other 
applications that provide information to users. 

[129] Although specific embodiments of the invention have been described, 
various modifications, alterations, alternative constructions, and equivalents are also 
encompassed within the scope of the invention. The described invention is not restricted to 
operation within certain specific data processing environments, but is free to operate within a 
plurality of data processing environments. Additionally, although the present invention has 
been described using a particular series of transactions and steps, it should be apparent to 
those skilled in the art that the scope of the present invention is not limited to the described 
series of transactions and steps. 

[130] Further, while the present invention has been described using a 
particular combination of hardware and software, it should be recognized that other 
combinations of hardware and software are also within the scope of the present invention. 
The present invention may be implemented only in hardware, or only in software, or using 
combinations thereof. 

[131] The specification and drawings are, accordingly, to be regarded in an 
illustrative rather than a restrictive sense. It will, however, be evident that additions, 
subtractions, deletions, and other modifications and changes may be made thereunto without 
departing from the broader spirit and scope of the invention as set forth in the claims. 



36 



